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1. Introduction 

Spin models of recurrent neural networks have been studied intensively within 
equilibrium and non-equilibrium statistical mechanics, especially after Hopfield P] 
emphasized their link with spin-glasses. In Hopfield's picture, the free energy minima 
of glassy systems (fixed-point attractors of the djTiamics) are given an information 
processing interpretation: in recurrent neural networks they represent stored items 
of information ('patterns'), whose locations in phase space are the result of suitable 
modification of the neuron interactions ('learning'). Spin glass theory, especially the 
replica method 0, was shown to be an efficient tool with which to study the equilibrium 
properties of Hopfield type models (or 'attractor neural networks') with full connectivity 
[HI E] . It was clear from the start that full connectivity is an undesirable simplification 
of biological reality, made only for mathematical convenience. However, it was also clear 
that solving neural network models with restricted range interactions in finite dimensions 
D was either pointless (no phase transitions for D = 1) or impossible (for D > 1). This 
dilemma prompted the study of so-called diluted models, where for each spin all but a 
randomly selected subset of size c of interactions are removed; such can be done either 
while preserving interaction symmetry, or asymmetrically (in the latter case detailed 
balance no longer holds). Different regimes for the scaling of c with the system size N 
were found to give different physics, and to require different mathematical techniques. 



Finite Connectivity Attractor Neural Networks 



2 



The early models of [SI E] correspond to c = A^. The Hopfield model with so-called 
extreme dilution (i.e. limjv^ooC"^ = lim^^oo c/N = 0, such as c = logA^) was studied 
in [Sj for the case of fully asymmetric dilution, and in for the case of symmetric 
dilution. Both in fully connected and in symmetrically extremely diluted networks 
close to saturation one finds a conventional replica theory a la with the familiar 
replica order parameter matrix. An alternative route away from full connectivity, which 
preserves both the potential for phase transitions and solvability, was followed in [71 [S] 
(the 1 + oo dimensional attractor networks). For recent reviews of the equilibrium and 
non-equilibrium statistical mechanics of recurrent neural networks see e.g. llUj. 

In this paper we study the as yet unsolved class of symmetrically diluted attractor 
neural networks with finite connectivity, where the average number c of bonds per neuron 
is finite, independent of the system size A^. For bond-disordered spin systems, the 
finite connectivity regime (which so far has been addressed only in the context of spin 
glasses [m 1121 HSl HH Uni HZI , error correcting codes [IHliin] and satisfiability problems 
|2() | l ^ l ^ l2lij) requires order parameter functions, which generalize the replica matrices 
of [2]. For finite c the replica symmetry breaking theory (RSB) is still under development 
[ini l21ll2Sll2ni [I3I2ZI • Here we apply the finite connectivity spin glass replica techniques 
to a general family of attractor neural networks (which includes the Hopfield model, but 
also the so-called clipped Hebbian synapses, as in e.g. ^B^), within the replica symmetry 
ansatz (RS). We obtain phase diagrams in the (a,T) plane, for arbitrary finite c, where 
a = p/c {p giving the number of stored patterns) and T is the temperature. These 
diagrams contain a paramagnetic phase (P), a recall phase (R), and a spin-glass phase 
(SG), all separated by second order transitions. We also analyze the RS ground state, 
and show how for c oo and arbitrary T one recovers the simpler results of [E]. The 
most surprising outcome of our calculations is the low values of the connectivity c, 
only barely exceeding the percolation threshold, which are required for the system to 
operate effectively as an attractor neural network; equivalent ly, the robustness of such 
information processing systems against excessive dilution and/or physical damage. 

2. Model Definitions 

We study Ising spin neural network models, with microscopic states defined by the N- 
neuron state vector cr = (ai, . . . , cta?) G {—1, 1}^. Here = 1 if neuron i fires, and 
(Tj = — 1 if it is at rest. Upon imposing standard Glauber- type dynamics, where the 
spins align stochastically to local fields of the form /ij(cr) = Jij'^j with symmetric 
interactions {Jij}-, these systems will evolve to thermodynamic equilibrium, described 
by the Hamiltonian H{cr) = Jij<^i<7j and the associated free energy 

F = -p-' log Z, Z = Y. e-^^^'^) (1) 

cr 
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The interactions are defined as a diluted and generalized version (in the spirit of [2H]) 
of the standard Hebbian recipe, with Cij G {0, 1} and Cij = Cji for all (i, j): 



The p vectors = , ■ ■ ■ , ^ 1}^ represent patterns to be stored in the 
system. The Cij G {0, 1} define the connectivity of the network, and act as quenched 
disorder. They are drawn randomly and independently from 

PiCr,) = ^5c.„l + [1 - ^]5c,,,0 (3) 

The average number of connections to any neuron is c. Averages over the realizations 
of the {cij} will be denoted as The function in Q need not be specified at 
this stage; for 0(x) = x one returns to the symmetrically diluted Hopfield model, for 
= y/p sgn(x) one finds diluted and clipped Hebbian synapses, etc. The relevant 
and nontrivial scaling regime is that where lim^y^ooP/c = a, with < a < oo. 

Models of the type ^ have so far been studied in regimes where limAr^oo c"^ = 0. 
Here we study the more complicated scaling regime where c = 0{N^). Our objective 
is to solve the model by calculating the disorder-averaged free energy per spin in 
the thermodynamic limit: / = — lim7v-*oo(/5A^)~^log Z. The replica identity logZ = 
lim„^o log allows us to write this in the standard manner as 



/ = - lim lim ^ log y e-^^^<^'''''^^^^-^^^^-''>? (4) 



Due to c = 0{N^), one can expand this expression for — >■ oo: 



f = — lim lim log exp 




In non-diluted disordered spin systems, the sum in the exponent would have been 
quadratic in the spin variables; there the free energy can be linearized by a gaussian 
transformation, leading to a single spin problem with the conventional replica order 
parameter matrix {q'a/?} |2]- In systems with finite connectivity, in contrast, one finds 
more complicated order parameters which encode higher order correlations between 
replicas ^T]. In these models it is more convenient to use an order parameter function 
[T^ IT^ ITij . describing the distribution of spin variables in the various replicas, from 
which all the conventional order parameters can be derived. 

3. Calculation of the Free Energy 

3.1. Replica Analysis and Sub-lattice Order Parameters 

To work out (0) we exploit the fact that for c = 0{N^), also p = 0{N^). This allows 
us to use the concept of sublattices jTHj (of which there are 2^): 

^^ = {^1^. = ^} P^ = \I^\/N (6) 
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We define cTj = {a}, . . . , cr"), and henceforth abbreviate averages over the sublattices as 
(/(^))^ = J2^P^f{^)- The trace in (0) can now be written as 



<T<T' 



E E ScT,cT.Sa',a, + 0{N') \ (7) 



We next introduce a spin distribution -P^(cr) within each sublattice, with cr 
{a^, . . . , 0""}, to be isolated upon inserting suitable ^-distributions into ((HI): 



1 = lY[i^dP^ia)5 



(8) 



From these distributions, one derives the more familiar types of observables such as 
replicated sublattice magnetizations m'i. and replicated pattern overlaps m'^": 



rrit 



<j]a 



(9) 



The introduction of integral representations for the delta functions in (jSJ generates 
conjugate variables P^(cr). The trace in (jZj) is now performed trivially, resulting in an 
integral to be evaluated by steepest descent: 



/ = — lim lim 



X 



cN 



log 



l[dP^{a)dP^{a) 



I ^ crcr' 



A4>{$.-^){(T-(T')_^ 



X exp < A^(log 



iPACT) 



1 



:5:(P^(a)P^(^))^ + (log 



iPACT) 



crcr' 



</.(£-^')(<T.<T')_^' 



k 

(10) 



Variation with respect to {P^(cr)} gives an equation with which to eliminate the 
conjugate order parameters, resulting in / = lim^^o extr{p^(cr)}/[{P^(cr)}], where 

e!0(|-|'){cr-cr')_^" 



/[• 



c 



crcr' 



(3n 



(lo^ 



(|-|')(cr.o-')_^' 



I' 



k 



'111 



Further variation, again with respect to P^(cr), then leads to the saddle point equation 
P^i^) = 7 ^ " r ...... ' ^' . (12) 



exp 1 


ciEa' P^'i^') 




k 


} 


E<T' exp 


[c{E(T"P^"i(T") 
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When deriving (|12p it first appears that one must also allow for adding to {P^(cr)} any 

vector in the kernel of the 2^+^^ x 2^+" matrix f/(^, cr; a') = exp[f • ^'){(t ■ a')] - 1. 
However, adding such elements is seen to generate new expressions for {-P^(cr)} which 
again solve ()12p: hence p2|) is always satisfied at the saddle-point. 

An alternative route for calculating the free energy will turn out to be more 
convenient for calculating zero temperature properties, such as ground state energy 
and entropy. Here we first vary (fTIHl with respect to P^(cr), resulting in / = 
lim„^o extr{p^(cr)}/[{P^(cr)}] subject to the constraints I]cr-P^(c) = 1 for all ^, with 

/[• • •] = - ^(( E ^|(-)^^'(- ) - 1]))^^' 

1 



era' 



+ ^iT.P^i^)^ogP^icT))^ (13) 

The conjugate order parameter functions -P^(cr) are now found to act as Lagrange 
multipliers, imposing the normalization constraints on the distributions P^(cr). 

3.2. RS Order Parameter Equations and Free Energy 

In order to take the n ^ limit we make the replica symmetric (RS) ansatz for the order 
parameters. One anticipates RS to be broken in the low temperature phases [Tl ITTl lU], 
but to hold up to the first phase transition away from the paramagnetic phase, as in 
the Viana-Bray model [llj. In the present context, RS is equivalent to assuming that 
the P^(cr) are invariant under permutations of the components of cr, i.e. only depend 

on the sum Y.a'^"- Upon defining effective fields [121, '^i^- ^« = i'^i) -^S 

order parameter functions can thus be written ^Tj as 

P^M = j,h W^ih)^^-^^ (14) 

Here W^{h) > and Jdh W^{h) = 1. One can write sub-lattice magnetizations and 
higher order observables in terms of the W^{h): 

^P^(cr)aV^...a'' = Jdh W^{h)tan\f{(3h) {a<(3<...<r) (15) 

The pattern overlaps m^" are now independent of a, m^" = m^, where 

= J dh W^{h) tanh{(3h))^ (16) 
We substitute flT^ into (O, and isolate the occurrences of J2a by inserting 

°° /■27r rlrh ^ 

. dm ^.^(^_^^,.) ^^^^ 

n.=-o^^0 27r 

After some mostly straightforward manipulations! one can then take the n —>■ limit 
and find (|12j) converting into: 

J dh W^{h)e'^''"' = exp |c ( J dh'W^ih') e-t^'^i>"'[t^'^i>(^'*')t'^'^Mf _ i )^,| (ig) 

I The only technical subtlety is the need to take n ^ for even n, to avoid in the denominator of 1)12(1 
tricky terms like logcos(TO) with to < 0. The same issue arises when calculating the RS free energy. 
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This is to hold for any real m. Provided the various integrals exist, we may now also 
put m —>■ im / (3 and carry out an inverse Fourier transform (following J7] ) , leading to 



imh 



27r 

X exp^c {ldh'Wt'{h') 



' ^J^ tanh-l [tanh(/3h') tanh(|fli(^-^'))] 



(19) 

These are the final equations from which to solve the RS order parameters, i.e. the TP 
effective field distributions W^{K). 

Upon inserting the RS ansatz ()14|1 into expression (jllll for the free energy, one can 
again take the limit n ^ (provided n is even). However, the calculation of / is found 
to be easier using the alternative expression (fT^ (see Appendix A for details). The 
result of the calculation is 



— {{J dhdh' W^{h)W^ih') log 



l+tanh(/3/i) tanh(ph') tanh(-0(^ ■ ^')) 

c 

1 



)) 



- ^((logcosh[^0(^ • ^')]))^|' -^(Jdh W^{h) log[2cosh(/3/i)])^ 



^{{JdhW^{h)\og 



P 



tanh\(3h) tanh2(^0(£-O) 



(20) 



In deriving ^K^ . which is reassuringly similar but not identical to the corresponding 
expression derived for the RS free energy of spin glasses with random interactions |13j . 
we have used W^{h) solving (fT9|) and being normalized (for every ^). 

As a simple test of our RS results, one may inspect the limit c — > oo for (j){x) = x. 
Here the interactions are of the Hopfield type, and, since p is finite, one should recover 
the equations describing the Hopfield model away from saturation. Expansion of the 
saddle-point equation (fTUj) in powers of 1/c, keeping only the (9((-)°) terms, results in 



W^{h) = 6{h-Y.e 



(21) 



Upon using this expression to calculate the overlaps = {C^^ J dhW^{h)taiih{ph))^ 
one indeed recovers the saddle-point equations of the Hopfield model away from 
saturation. Similarly, (pUj) reduces to the correct corresponding free energy. 



4. Analysis of Phase Transitions 

4.I. Expansion of the saddle-point equations 

The paramagnetic state W^{h) = 6{h) V| always solves (fTT^ . As the temperature is 
lowered from T = oo, we expect other solutions to bifurcate away from the paramagnetic 
one. In finite connectivity spin glasses, phase transitions were found upon expanding the 
RS free energy in the order parameters (fT^ near the paramagnetic phase TT^ . A similar 
strategy can be applied here, following [12]. We assume that, close to the transition, 
Jdh Wd{h)h^' = 0{e^) with < e -C 1, and we expand both sides of p8|) in powers of e. 
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This can be done self-consistently, for all al orders in e. Identification of the lowest two 
orders e and is then found to give, respectively 

J dh W^{h)h = c{jdh W^ih)htanh[^(P{$, ■ (22) 

J dh W^{h)h^ - Jdh W^{h)h ^ = c{Jdh W^ih)hHanh^[(^(j){^ ■ |')])^' (23) 

There are hence two types of transitions away from the paramagnetic (P) state. The first 
corresponds to the lowest order being e; such transitions are marked by the appearance 
of nonzero solutions of (j22I)- The second type have as lowest non-zero order; these 
transitions are marked by non-trivial solutions of with Jdh W^{h)h = 0. We 
conclude that these transitions are marked by the highest temperature for which the 
two 2^ X 2^ matrices and Q^fi', respectively, have an eigenvalue equal to 1: 



It follows from ()16|) that bifurcation of M eigenvectors corresponds to a transition 
towards a retrieval state (R), with nonzero pattern overlaps, whereas bifurcation of 
Q eigenvectors corresponds to a transition towards a spin-glass state (SG). 



cp^,tanh2[^0(|.O] (24) 



4-2. Transition Temperatures for Random Patterns 

For independently drawn random patterns, where p^ = 2^^ for all ^, both matrices 
in ()24|1 are symmetric and can be diagonalized by exploiting symmetries. The desired 
eigenvectors have in fact already been calculated in j2H] (in a different context). They 
are found to be universal for all 2^ x 2^ symmetric matrices whose entries depend only on 
^ ■ i.e. U^^' = ■ ^'). With each of the 2^ index subsets S* C {1, 2, . . . ,p} one can 
associate an eigenvector vs, defined as vs{$,) = Yln^s^^- The eigenvector corresponding 
to the empty set is defined as v${$,) = 1 for all ^. One easily verifies that this gives all 
eigenvectors of U, with corresponding eigenvalues 

^s = j:u{tnEe (25) 

The eigenvalues depend only on the size 15*1 of the subset. Application of (j25|) to our 
matrices M and Q gives the following eigenvalue spectra: 



M 



A 



Clearly, lim^_,o A 



nes 



tanh[^0(5:e.)]). 



A^ 



ties 



tanh^[- 

c 



(Ee.)])/ 



M 
S 



lim^^o Ac 



0. For monotonically non-decreasing functions 



0(x), the largest M eigenvalue is found for \S\ 
corresponds to 5* = 0: 



A 



M 



I ((Entanh[^0(5:r)])^ 



Alx = c(tanh^F0(En]) 



1, whereas the largest Q eigenvalue 

(26) 
(27) 
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Figure 1. Phase diagram of the finite connectivity Hopfield model, where (p{x) — x. 
Connected markers give the P^R and P^SG transition temperatures in the (a, T) 
plane (where a — p/ c), iov c — 2 (□), c — i (<)), c = 10 (+), whereas the c = 100 
transition is indicated by a solid line. The transition line for c = 100 is nearly 
indistinguishable from the corresponding c —> oo line segments T — \ (for a < 1) 
and T = ^/a (for a > 1). 



This leads us to the following equations for the P— i^R and P^SG transition lines: 

P ^ R : - r ) (p - 2n) tanh[^(/.(p - 2n)] = 1 (28) 

P^SG: c2-^'X^ K)tanh2[^0(p-2n)] = 1 (29) 

Equations ()28l29p will generally have to be solved numerically. 

We may finally inspect the limit c oo, with a = p/c fixed, where we expect to 
find a generalization of the results in The central limit theorem now allows us to 
replace Su=i by a zero-average and unit- variance Gaussian variable, so 

112 

simplify to (provided the limits exist, and with Dy = (27r)~2e~2?' ): 



il^^-x = l^jDyy\uni^^ tanh[^</.(|/^)]| (30) 
iHS>^-x =l^jDy Jim L t^nh'A{y^)]\ (31) 



4-3. Application to Specific Synaptic Kernels 



We now have to make explicit choices for the kernel 4>{x) in our expression for the 
interaction matrix, viz. Jij = ^ 0(^j ■ ^j). We first inspect the finite connectivity 
Hopfield model Jij = ^ ■ ^j- Here we have </'(x) = x, and the c oo expressions 
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Figure 2. Phase diagram of the finite connectivity chpped Hopfield model, where 
(j>{x) — y/p sgn(x). Connected markers give the P— >R and P^SG transition 
temperatures in the {ot,T) plane (where a — p/c)^ for c = 2 (□), c — i (<^), c — IQ 
(+), whereas the c = 100 transition is indicated by a solid line. The transition line for 
c = 100 is again nearly indistinguishable from the corresponding c ^ oo line segments, 
here given by T = \/27^ (for a < ^2/7r) and T = ^/a (for a > a/S/tt). 



for the P— i^R and P— >SG transition temperatures (jHOpHlj) reduce to Tr = 1 and 
TsG = (i^ accordance with the extreme dilution results in [F^). In addition one 
finds that for c — > oo the effective field distributions W^{h) become Gaussian, with mean 

7^^ = {Jdh W^' tanh(/5/i)(^-^'))^' and variance = a{Jdh W^' tanh\f3h){^-^')) ^> -hi, 
from which one immediately recovers the RS order parameter equations of [H] (describing 
states with a single 'condensed' pattern, for the scaling regime c — >■ oo with c/N 0): 

m = J Dy tanh.[f3 {m + y^/aq)] q = J Dy tanh^ [l3{m + y^/aq)] 

The results of solving ()28p29j) numerically for finite c are shown in figure ^ Note that 
a = p/c E {^, f , f , • • •}, so only the actual markers in figure [T] correspond to physically 
reahzable parameter values (the connecting line segments just provide a guide to the 
eye). We observe that for c = 100 the transition lines away from the paramagnetic phase 
are already nearly identical to those corresponding to c — oo. For c = 2, the system 
remains in a paramagnetic phase up to T = for a = 1. Here, interestingly, for a = 0.5 
(i.e. p = 1) there is a transition to a nonzero retrieval overlap at finite T. This is not 
what one would find in e.g. a 1 — d Ising chain, in which the connectivity is c = 2, but 
where there is no phase transition at finite T; the difference is due to the fact that in 
our present model the connectivity equals 2 only on average. Note that the value c = 1 
corresponds to the percolation transition ^21- We will turn to the location of the more 
complicated R— >SG transition in a subsequent section. 
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Figure 3. Phase diagram of a model intermediate between the previous two, with 
(j){x) = X for |a;| < y/p and (f){x) = ^ sgn(a;) for > y/p. Connected markers give the 
P^R and P^SG transition temperatures in the (a, T) plane, for c = 2 (□), c = 3 (<)), 
c = 10 (+), whereas the c = 100 transition is indicated by a solid line. The transition 
line for c — 100 is again nearly indistinguishable from the corresponding c — > cxd line 
segments, here given by T = Erf(l/V2) (for small a) and T = [a(l - ./ifi^)]^ (for 
large a). 



Our second application is obtained upon choosing a finite connectivity version of 
tlie so-called clipped Hebbian synapses: Jij = ^a/p sgn(^j • ^j) (the specific scaling 
with ensures a proper limit c —>■ oo). Here 4>{x) = ^ sgn(x), and the c ^ oo 
expressions for the P— i>R and P— s>SG transition temperatures ()3()pHl|) now reduce to 
Tfj = \J'2,/tx and Tsg = ^/ci- The results of solving ()28I29|) numerically for finite c 
are shown in figure El As before, only the actual markers in figure El correspond to 
physically realizable parameter values. As was the case for full connectivity |28j, also 
with finite connectivity one finds, surprisingly from an information processing point of 
view, that the differences between Hebbian and clipped Hebbian synapses are only of a 
quantitative nature; limited largely to a rescaling of critical temperatures. 

Our third and final finite connectivity network example is one which interpolates 
between Hebbian and clipped Hebbian synapses: (f){x) = x (i.e. Hebbian) for |x| < y/p, 
and 0(x) = ^/p sgn(x) (i.e. clipped Hebbian) for |x| > ^/p. The value y/p is found 
to be the natural and most interesting scaling for the cut-off point in this definition. 
The c oo expressions for the P— and P— >SG transition temperatures ()30I31|) now 
reduce to Tr = Erf (1/^2) and Tsg = [«(! - ^2/7re)]i The results of solving ^MI^ 
numerically for finite c are shown in figure El Again only the markers correspond to 
physically realizable parameter values. The apparent irregularities in the transition lines 
are due to the cut-off in the function 0(x) at |£ • = y/p. 
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5. RS Ground State of the Finite Connectivity Hopfield Model 

For large a, one expects replica symmetry to break at low temperatures. In contrast 
to full connectivity models, for finite connectivity models RSB theory is still under 
development. Here we will therefore analyze as yet only the T — limit of our RS 
equations. Apart from giving exact statements at least for small a, a calculation of 
the RS ground state entropy leads for large a to a bound for the location of the RSB 
transition in the phase diagram, since the latter must come before the zero entropy line. 



5.1. RS Order Parameters atT = 



We follow closely the RS ground state analysis carried out for finite connectivity spin 
glasses ^21 ^j- At T = there are no thermal fiuctuations, so the effective fields in 
()14|) are identical to the true local fields. The latter can take only discrete values, due 
to the finite number of connections per spin, hence each W^{h) is a sum of delta peaks: 



(32) 



e=-oo 



Upon taking the limit /3 ^ oo in ^T^ . using lim^_,oo tanh~^[tanh(/9a;) tanh(/?|/)] = 
+ y\ — — y\, one verifies by substitution that (jH^ is indeed a solution. Insertion 
into the right hand side of (fT^ gives (with 6nm = 1 — ^nm) '■ 

£'=1 



dy 



W^ih)= /^e-^^^exp I c 



27r 




+ E [KAne-^^-^' + K.i-ne-'^^-^'] - [1 - K,iO)] 5,,, J ) (33) 

Upon expanding the exponent in the right-hand side, we would indeed recognize a sum 
of delta peaks. Integrating of both sides over an infinitesimally small interval around 
h = i/c leads to equations for the factors K^{i): 



K^{i) = lim J ^ sin(z)e-^ exp ^ ^| '^E [^^K^Oe"^'^"^^'^'^ + A'^.(-f )e-^^s"(^-^') 



/ 



+ E [K^ine^^^-^' + K^i-ne-^^^-^'] - [1 - K^iO)] 5^.^. / 



(34) 



17 



For e — > 0, the periodic part of the integral (involving z/ce) oscillates rapidly 
and decouples from the non-oscillating part. We note that (jMj) is of the form 
lim^^o sin(z)/(2/e), where f{z + 2ti) = f{z). The function / may be written as 
a Fourier series, f{z/e) = J2'^=-oo'^ne^^^^^- Consequently: 



lirn /— sin(2;)/(-) = /— sin(2;) + J2^n hm /— si 

e^Oj TTZ e J TTZ ~i e^O J Tiz 



dz 



' dz 



sm I ez) = Oo 



Finite Connectivity Attractor Neural Networks 

Since = {2tt)^^ JZ^dcj) /(0), expression (j21|) can be simplified to 



12 



-c([i-/c^,(o)]5. r dd) 



X COS 



exp 



V 



+ c 



sin(£'0) sgn(^ ■ 



+ ^ [/O.(0-%(-0]sin(^-C^ 



^5 



Solutions to this set of equations can be found numerically, via iteration. 



(35) 



5.2. RS Entropy atT = 

We next calculate the RS zero temperature entropy per spin sq, by expanding the free 
energy per spin up to order T: 

f = fT=o-soT + OiT^) (36) 

Care is needed in taking the temperature derivative of the free energy, in view of the 
normalization constraint on the order parameters W^{h). One generally has 



df df[. 



dT dT 



saddle 



saddle 



dW^,{h') 
df 



(37) 



where W* denotes the saddle-point of /[. . .]. In contrast to unconstrained extremization, 
the functional derivative in the right hand side of ()37j) need not vanish, since 
extremization of / is restricted to the subspace in which jdh W^{h) = 1: 



5fl^ 6 



Jdh' W^ih') - 1 



(38) 



Here {A^'} are Lagrange multipliers. As a consequence, we find 6f[. . ■]/SW^{h) = 
at the saddle point (i.e. independent of the field h), and hence 



df df[. . .] 



dT dT 



saddle ^' 



df[. . .] 



dT 



(39) 



saddle 



For the purpose of calculating sq, we may thus simply insert the T = saddle-point 
()32p35|l into our expression (j20|l for the free energy, giving: 

c ^ , , . , tanh(— ( 

c c c 



So= \unf]'^{{ -- Y ^£(^)%(Olog[l + tanh(^)tanh(^)tanh(^^-0 
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Figure 4. Zero temperature phase diagram of the finite connectivity Hopfield model, 
in RS approximation and for odd p (note: p = ac). Connected diamonds (<^): the 
R^SG transition line. Dashed horizontal line: the c ^ oo location for the R^SG 
line, Qfc — S/tt, as known from 



log[l + tanh(^) tanhf^l ■ ^')] + log[l - tanh(^) tanh(^| ■ |')] 

c c c c 



log cosh(^^ ■ - ^ E KM log[2 cosh(^)] 
Zfj c p c 



€1' 



log2 (^( E {K^H^-a) + K^i\^-^'\)-[K^ii) + K^i-m^-l5i,o]] 



+ -[21og2-log3] (( [K^{-\^.^'\) + K^{\^.a 



This result is reminiscent of the corresponding expression found for spin glasses in |29j . 



5.3. Zero Temperature Phase Transitions 

We next try to solve the coupled equations ()35p . for randomly drawn patterns, at zero 
temperature. We know that in the case of perfect retrieval of a pattern /i, half of the 
sublattice magnetizations (the ones with = 1) must be 1 and the other half must 
be —1. At T = all spins align with their local fields, so it follows from ()32|1 that 
K^{i) = K{^^t) for some set of non- negative numbers K{i). We will now choose the 
property K^{i) = K{C,'^i) for all ^ G {—1,1} and all integer i as an ansatz. Upon 
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restricting ourselves for simplicity to odd values of p , our equations subsequently 
simplify to 

K{i) = e-^[i-^(o)] r dl ^cA(MK}) ^os[_^0 + c5(0, {K})] (41) 
with the abbreviations 

2m— p~l 

A{<p, {K}) = 2-(^-^) ( ) E [^W + cos(£0) 

oo 

cos[(p-2m)0] l^W+^i-^)] (42) 

m) = X: f ^ 1 sin[(p - 2m)0] f] [K{i) - K{~i)] (43) 

m=o ^' \ m y e=\p-2m\ 

The result of solving PT|) numerically, and subsequently detecting the R— i>SG transition 
(which is found to be first order) is shown in figure |3] in the {p, a) plane. The R^SG 
transition line should for c — cxd (i.e. p —>■ oo with fixed a) approach the value ac = 2/ti 
( ~ 0.637), in accordance with [HI; the latter limit is shown in the figure as a horizontal 
dashed line, which is indeed seen to be approached by our R— i>SG line as p — oo. As 
usual for the present type of system, we concentrated on bifurcation points. Due to the 
extensive energy barriers between the ergodic sectors in attractor neural networks, on the 
important time-scales local stability is more important than thermodynamic stability, 
so the thermodynamic transition lines (based on comparing values of free energies) are 
not relevant. Numerical evaluation of the free energy of the candidate solution in fact 
reveals that the bifurcation lines coincide with the thermodynamic transitions. 

Figure lU shows re-entrance phenomena. It should be noted that an increase of 
p for fixed a implies a simultaneous increase of the number of stored patterns (with a 
detrimental effect on recall quality) and of the connectivity c (expected to have a positive 
effect on recall quality); the non-monotonic appearance of the R^SG line reflects the 
competition of these opposite tendencies. 

Finally, for the present type of zero temperature solution, our expression for the 
RS zero temperature entropy reduces to 



'° K{Q) 



log 2 




K{2m -p) + K{p - 2m) 



-[m+K{-i)][i~\5,,,] 



+ c2-^'(l-4^) Y. ["^ \[K{2m-p) + K{p-2m)Y (44) 




21og2' 

Numerical evaluation of this expression reveals that the RS entropy at T = is never 
zero, for any combination of a and p. This at first sight somewhat surprising result is 
in fact in accordance with a similar observation made recently for finite connectivity 
spin-glasses in [21]. It does, however, not imply that replica symmetry is not broken at 
low temperatures (which we know must happen for c — cx) jH]). 
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6. Discussion 

In this paper we have shown how the mathematical framework of finite connectivity 
spin glass replica theory (involving order parameter functions in replica space, rather 
than matrices) can be combined with the concept of sub-lattices in order to solve a 
large class of finite connectivity Hopfield-type attractor neural network models near 
saturation. So far such networks appear to have been studied only in the two technically 
simpler regimes of full connectivity and so-called extreme dilution, both of which involve 
a diverging number of bonds per spin c in the thermodynamic limit. We have restricted 
ourselves to a replica symmetric (RS) analysis. The replica symmetric theory is found 
to lead to 2^ coupled integral equations (fT^. whose solutions give the effective local field 
distributions in each of the 2^ sublattices of the system. Here p denotes the number of 
stored patterns. In the limit c oo our equations are shown to reduce to the theory 
of so-called extremely diluted attractor neural networks, as expected. For T and 
below a certain critical value for a one should expect replica symmetry no longer to hold 
(it has been shown for c — ^> oo in [B^); going beyond replica symmetry would require 
extending the theory to include one or more steps of replica symmetry breaking, similar 
to the finite connectivity spin glass calculations in [ini Ell 12^1 UEi HZl 121] ■ 

As is usual for attractor neural networks near saturation, our phase diagrams exhibit 
three phases: a paramagnetic phase (P), a retrieval phase (R), and a spin-glass phase 
(SG). We have calculated, for arbitrary values of the number of bonds c per spin and 
a large class of Hebbian type synaptic kernels, the (second order) P— >R and P— *^SG 
transition lines in the (a,T) phase diagram. For the main member of our model class, 
the finite connectivity Hopfield model, we have also calculated the (second order) R— >SG 
transition line at T = 0. We find that the values of the average connectivity c needed 
for the system to function as an attractor neural network are surprisingly small, barely 
exceeding the percolation threshold, even for clipped Hebbian synapses (where each 
bond carries only one bit of information). Figures ^ to El for instance, underline that 
already for c = 3 the phase diagrams differ only in a relatively modest sense from those 
corresponding to c ^ oo. This underlines the robustness of recurrent neural networks 
as information processing systems. 

Apart from the obvious extension of our present work, the inclusion of RSB solutions 
of our saddle-point equations and calculation of AT lines ^], it would also be an 
interesting challenge to attempt a dynamical solution. Within the generating functional 
analysis formalism of [31 this would for finite connectivity systems involve an effective 
single spin problem, with order parameters describing single-spin path probabilities and 
the effect on these path probabilities of single-spin external field perturbations. 
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In this section we calculate the RS free energy per spin, upon inserting (fT^ into (fT!?jl . 
In doing so we will use the short-hands: 



7r7 



'Tt/2 TT 



log[cos(0)]. cos(2m(/ 



m = 0,±1,±2, 



They obey: 



JZam 



log cosh (a) 



(A.l) 



Taking the limit n — is again found to impose the need to restrict ourselves to even 
values of n. One then obtains, after some algebra, for the entropic term in (jl3|) : 

lini-l(VP.(^)logP.(^)).= hm-^ V r^e^™™ /d/ie^'^'" 



m=— oo 



cosh(/3/i) 



oo „ p „ 

^mJdh{W^ih)e'^'^"'\og Jdh'W^ih')e'^'^'"^p^ 



-Jdh (W^^(/i)log[2cosh(/3/i)])^ 



(A.2) 



Similarly, for the energetic term U in (fT^ one finds: 



^Jdhdh' {{W^{h)W^ih') log 



J2 e-.^ii^)'^<^'+P[h^+h''^'\ 



)) 



+ -7^ 



^jdh (W|(/i)logcosh(/?/i))| 



c 



dhdh' {{W^{h)W^{h') log 



1 + tanh(/5/i) tanh(/?/t') tanh(-0(| • ^)) 

c 



";((logCOsh[^0(^O]))^^' 



(A.3) 



2/3 ■ - -c 

We use fll8|) to simplify the entropic term ()A.2|) . and add the latter to the energetic 
term ()A.3|) to obtain the following expression for the RS free energy per spin: 



/rs= --Jdh (W^^(/i)log[2cosh(/?/i)])^-^((logcosh[^</.(^-0])) 



c 



dhdh' {{W^{h)W^{h') log 



1 + tanh(/3/i) tanh(/3/i') tanh(-0(| ■ ^')) 



^1' 



+ ^ Y 1^ J dhdh' {{W^{h)W^ih')e^'"'"' 'e^^'--^-'i^--HPh')t.nH^c^it^'))] _ ^ 



At this stage we may use relation ()A.1|) to carry out the summation over m. 
After some further re- arrangement of terms, and application of the simple identity 
cosh[tanh^^(y)] = (1 — y^)^^/^, one then arrives at the final result (PUj). 



